Run number 1000

To produce a set of diagnostic plots that will be included in a report. Please note that these plots are just meant to provide an example of what could be created and how. They are not an exhaustive list of every possible plot and were chosen with the project aims in mind.

Set up

Model location

Define modelName and path to the model directory (MODEL_DIR).

Figure location

If saving figures out to pdf, define where those pdfs should be saved to. Here the figures are saved to deliv > figure > model_run_number

Model details

A summary of high-level model details.

Read in the model details using read_model. Details stored in the mod object can be used to identify the location of the source data (used in $DATA) - to see how this is done look at the bbr::get_data_path() and bbr::build_path_from_model() helper functions.

── Status ─────────────────────────────────

• Finished Running

── Absolute Model Path ───────────────────────────

• /data/NMBayesTutorial/model/pk/1000

── YAML & Model Files ───────────────────────────

• /data/NMBayesTutorial/model/pk/1000.yaml

• /data/NMBayesTutorial/model/pk/1000.ctl

── Tags ──────────────────────────────────

• one-compartment + absorption

• ETA-CL

• ETA-KA

• ETA-V

• CLWT-allo

• V2WT-allo

• QWT-allo

• V3WT-allo

• proportional RUV

• BAYES

Read output

Parameter estimates

Compute summaries of posterior distributions as well as some diagnostics.

Bulk effective sample size (ESS) is a measure of sampling efficiency for the location of the distribution, while Tail ESS is a measure of sampling efficiency for the tails (5% and 95% quantiles) of the distribution. Higher values indicate greater sampling efficiency. A very rough rule of thumb is to aim for at least 400 for each parameter.

R-hat is a convergence diagnostic that compares the between- and within-chain variances of model parameters. Values close to 1 indicate that the chains have converged to similar distributions. Aim for less than about 1.05 for all parameters.

Summary of model parameter estimates.
variable mean median sd mad q5 q95 rhat ess_bulk ess_tail shrinkage
THETA[1] 0.475 0.474 0.0647 0.0651 0.371 0.583 1.00 3245 7505 NA
THETA[2] 4.10 4.10 0.0284 0.0285 4.05 4.14 1.00 4470 11807 NA
THETA[3] 1.10 1.10 0.0336 0.0333 1.05 1.16 1.00 37693 36437 NA
THETA[4] 4.23 4.23 0.0231 0.0230 4.19 4.26 1.00 2072 4686 NA
THETA[5] 1.30 1.30 0.0369 0.0368 1.23 1.35 1.00 727 1257 NA
OMEGA[1,1] 0.257 0.252 0.0576 0.0557 0.173 0.360 1.00 3459 6688 21.0
OMEGA[2,1] 0.0778 0.0763 0.0212 0.0207 0.0456 0.115 1.00 6106 12457 NA
OMEGA[2,2] 0.0903 0.0893 0.0122 0.0118 0.0719 0.112 1.00 13779 24992 9.02
OMEGA[3,1] 0.138 0.136 0.0270 0.0265 0.0968 0.185 1.00 8595 18059 NA
OMEGA[3,2] 0.0742 0.0734 0.0128 0.0124 0.0547 0.0966 1.00 21608 33149 NA
OMEGA[3,3] 0.174 0.173 0.0201 0.0196 0.144 0.210 1.00 36108 36486 2.09
SIGMA[1,1] 0.0395 0.0395 0.00112 0.00113 0.0377 0.0414 1.00 23292 31561 NA

LOO-CV

PSIS-LOO CV is leave-one-out (LOO) cross-validation for Bayesian models using Pareto smoothed importance sampling (PSIS). This is used for model comparison, similar to the -2LL objective function for non-Bayesian models.

## 
## Computed from 40000 by 160 log-likelihood matrix
## 
##          Estimate     SE
## elpd_loo  27770.9  961.6
## p_loo      2919.7   43.4
## looic    -55541.7 1923.1
## ------
## Monte Carlo SE of elpd_loo is 0.1.
## 
## All Pareto k estimates are good (k < 0.5).
## See help('pareto-k-diagnostic') for details.

Read in data

Model output (EPRED, IPRED, NPDE, EWRES, ETAs) is read in from either the output of simulations (see ?bbr.bayes::nm_join_bayes) or from NONMEM output only. When only NONMEM output is used, medians of these values across all chains are calculated.

After reading in the NONMEM dataset and the output dataset they’re joined by a NUM column. This assumes that a row number column (called NUM) was included during data assembly. The idea here is that in NONMEM, you table just NUM and none of the other input data items. They all will get joined back to the NONMEM output … even character columns.

The data used in the diagnostic plots has been filtered to only include the observations (i.e. EVID==0). Note that further modifications maybe needed, for example, if BLQ data was included in the model or if the DV was log-transformed. The dataset also converts the categorical covariates of interest to factors using the yspec_add_factors function and details described in the spec file.

The id subset gets the first record per ID. This would usually be the baseline value but consider filtering on a baseline flag if available. Also, if the model includes inter-occasion variability (IOV), the occasion variable should be included within the distinct function.

Convergence diagnostic plots

These plots assess whether all of the chains have converged to a single, stationary distribution.

Trace and density plots

## [[1]]

## 
## [[2]]

## 
## [[3]]

## 
## [[4]]

## 
## [[5]]

## 
## [[6]]

## 
## [[7]]

## 
## [[8]]

## 
## [[9]]

## 
## [[10]]

## 
## [[11]]

## 
## [[12]]

Pairs

Look for any strong correlation between parameters, indicating possible over-parameterisation.

ESS vs sample

Plots Bulk and Tail ESS versus iteration to check that ESS increases linearly.

## [[1]]

## 
## [[2]]

## 
## [[3]]

## 
## [[4]]

## 
## [[5]]

## 
## [[6]]

## 
## [[7]]

## 
## [[8]]

## 
## [[9]]

## 
## [[10]]

## 
## [[11]]

## 
## [[12]]

ESS vs quantile

Plots Bulk and Tail ESS versus quantile to better diagnose areas of the distributions that the iterative algorithm fails to explore efficiently.

## [[1]]

## 
## [[2]]

## 
## [[3]]

## 
## [[4]]

## 
## [[5]]

## 
## [[6]]

## 
## [[7]]

## 
## [[8]]

## 
## [[9]]

## 
## [[10]]

## 
## [[11]]

## 
## [[12]]

General diagnostic plots

The following plots assume that the preferred x-axis labels are defined here.

DV vs population and individual predictions

Create plots of DV vs population and individual predictions for the full dataset and stratified by renal function and hepatic function.

Population predictions are medians of 2000 simulated values incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution. Individual predictions are medians of 2000 simulated values incorporating conditional estimates of individual parameters and include within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.

NPDE plots

Normalized prediction distribution errors (NPDE) are Monte-Carlo generated diagnostics, using 2000 simulations incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.

NPDE vs population predictions, time and time after dose.

Population predictions are medians of 2000 simulated values incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.

NPDE vs continuous covariates

NPDE vs categorical covariates.

NPDE density histogram

EWRES vs population predictions, time and time after dose

Expected weighted residuals (EWRES) are Monte-Carlo generated residuals, using 2000 simulations incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.

Population predictions are medians of 2000 simulated values incorporating between- and within-subject variability, as well as uncertainty in population parameter estimates via sampling from the posterior distribution.

EWRES qq and density plot

EBEs-based diagnostics

ETAs are medians of 2000 posterior ETAs across the 4 chains.

ETA pairs plot

Continuous covariate plots

These plots uses the yspec to automatically rename the axis labels.

ETA vs continuous covariates

Note that here we use a function that maps over the ETAs (not the covariates) because the purpose of these plots was to determine whether there were any trends in the covariates for a given ETA. This may need to be edited to address different study specific questions

## [[1]]

## 
## [[2]]

## 
## [[3]]

Continuous covariate pairs plot

Categorical covariate plots

These plots uses the yspec to automatically rename the axis labels.

ETA vs categorical covariates

Note that here we use a function that maps over the covariates (not the ETAs) because the purpose of these plots was to determine whether there is any difference in the distribution of ETAs across studies, dosing groups and disease states. This should be updated to reflect the questions you’re trying to address.

## $STUDY

## 
## $RF

## 
## $CP

## 
## $DOSE

Session details

It is considered good practice to include these details at the end of all rmd scripts

Sys.getenv("AMI_NAME")
## [1] "metworx-22.09-20220915120953-e92bea3c"
sessioninfo::session_info()
## ─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.1.3 (2022-03-10)
##  os       Ubuntu 18.04.6 LTS
##  system   x86_64, linux-gnu
##  ui       RStudio
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2023-12-19
##  rstudio  2022.02.4+500.pro1 Prairie Trillium (server)
##  pandoc   2.17.1.1 @ /usr/lib/rstudio-server/bin/quarto/bin/ (via rmarkdown)
## 
## ─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────
##  !  package        * version    date (UTC) lib source
##     abind            1.4-5      2016-07-21 [1] CRAN (R 4.1.3)
##     assertthat       0.2.1      2019-03-21 [1] CRAN (R 4.1.3)
##     backports        1.4.1      2021-12-13 [1] CRAN (R 4.1.3)
##     bayesplot      * 1.10.0     2022-11-16 [1] CRAN (R 4.1.3)
##     bbr            * 1.7.0      2023-11-17 [1] MPNDEV (R 4.1.3)
##     bbr.bayes      * 0.1.1.8001 2023-12-14 [1] MPNDEV (R 4.1.3)
##     bit              4.0.5      2022-11-15 [1] CRAN (R 4.1.3)
##     bit64            4.0.5      2020-08-30 [1] CRAN (R 4.1.3)
##     bslib            0.5.1      2023-08-11 [1] CRAN (R 4.1.3)
##     cachem           1.0.8      2023-05-01 [1] CRAN (R 4.1.3)
##     callr            3.7.3      2022-11-02 [1] CRAN (R 4.1.3)
##     checkmate        2.2.0      2023-04-27 [1] CRAN (R 4.1.3)
##     cli              3.6.1      2023-03-23 [1] CRAN (R 4.1.3)
##     codetools        0.2-19     2023-02-01 [1] CRAN (R 4.1.3)
##     colorspace       2.1-0      2023-01-23 [1] CRAN (R 4.1.3)
##     cowplot        * 1.1.1      2020-12-30 [1] CRAN (R 4.1.3)
##     crayon           1.5.2      2022-09-29 [1] CRAN (R 4.1.3)
##     data.table       1.14.8     2023-02-17 [1] CRAN (R 4.1.3)
##     diffobj          0.3.5      2021-10-05 [1] CRAN (R 4.1.3)
##     digest           0.6.33     2023-07-07 [1] CRAN (R 4.1.3)
##     distributional   0.3.2      2023-03-22 [1] CRAN (R 4.1.3)
##     dplyr          * 1.1.3      2023-09-03 [1] CRAN (R 4.1.3)
##     evaluate         0.22       2023-09-29 [1] CRAN (R 4.1.3)
##     fansi            1.0.5      2023-10-08 [1] CRAN (R 4.1.3)
##     farver           2.1.1      2022-07-06 [1] CRAN (R 4.1.3)
##     fastmap          1.1.1      2023-02-24 [1] CRAN (R 4.1.3)
##     forcats        * 1.0.0      2023-01-29 [1] CRAN (R 4.1.3)
##     fs               1.6.3      2023-07-20 [1] CRAN (R 4.1.3)
##     furrr          * 0.3.1      2022-08-15 [1] CRAN (R 4.1.3)
##     future         * 1.33.0     2023-07-01 [1] CRAN (R 4.1.3)
##     future.apply     1.11.0     2023-05-21 [1] CRAN (R 4.1.3)
##     generics         0.1.3      2022-07-05 [1] CRAN (R 4.1.3)
##     GGally           2.1.2      2021-06-21 [1] CRAN (R 4.1.3)
##     ggplot2        * 3.4.4      2023-10-12 [1] CRAN (R 4.1.3)
##     globals          0.16.2     2022-11-21 [1] CRAN (R 4.1.3)
##     glue           * 1.6.2      2022-02-24 [1] CRAN (R 4.1.3)
##     gridExtra        2.3        2017-09-09 [1] CRAN (R 4.1.3)
##     gridGraphics     0.5-1      2020-12-13 [1] CRAN (R 4.1.3)
##     gtable           0.3.4      2023-08-21 [1] CRAN (R 4.1.3)
##     here           * 1.0.1      2020-12-13 [1] CRAN (R 4.1.3)
##     hms              1.1.3      2023-03-21 [1] CRAN (R 4.1.3)
##     htmltools        0.5.6.1    2023-10-06 [1] CRAN (R 4.1.3)
##     inline           0.3.19     2021-05-31 [1] CRAN (R 4.1.3)
##     jquerylib        0.1.4      2021-04-26 [1] CRAN (R 4.1.3)
##     jsonlite         1.8.7      2023-06-29 [1] CRAN (R 4.1.3)
##     knitr            1.44       2023-09-11 [1] CRAN (R 4.1.3)
##     labeling         0.4.3      2023-08-29 [1] CRAN (R 4.1.3)
##     lattice          0.21-9     2023-10-01 [1] CRAN (R 4.1.3)
##     lifecycle        1.0.3      2022-10-07 [1] CRAN (R 4.1.3)
##     listenv          0.9.0      2022-12-16 [1] CRAN (R 4.1.3)
##     loo            * 2.6.0      2023-03-31 [1] CRAN (R 4.1.3)
##     lubridate      * 1.9.3      2023-09-27 [1] CRAN (R 4.1.3)
##     magrittr         2.0.3      2022-03-30 [1] CRAN (R 4.1.3)
##     Matrix           1.6-1.1    2023-09-18 [1] CRAN (R 4.1.3)
##     matrixStats      1.0.0      2023-06-02 [1] CRAN (R 4.1.3)
##     mclust           6.0.0      2022-10-31 [1] CRAN (R 4.1.3)
##     mgcv             1.9-0      2023-07-11 [1] CRAN (R 4.1.3)
##     mrggsave       * 0.4.5      2023-11-24 [1] MPNDEV (R 4.1.3)
##     mrgsolve       * 1.2.0      2023-11-04 [1] MPNDEV (R 4.1.3)
##     munsell          0.5.0      2018-06-12 [1] CRAN (R 4.1.3)
##     nlme             3.1-163    2023-08-09 [1] CRAN (R 4.1.3)
##     npde             3.4        2023-06-15 [1] CRAN (R 4.1.3)
##     parallelly       1.36.0     2023-05-26 [1] CRAN (R 4.1.3)
##     patchwork      * 1.1.3      2023-08-14 [1] CRAN (R 4.1.3)
##     pillar           1.9.0      2023-03-22 [1] CRAN (R 4.1.3)
##     pkgbuild         1.4.2      2023-06-26 [1] CRAN (R 4.1.3)
##     pkgconfig        2.0.3      2019-09-22 [1] CRAN (R 4.1.3)
##     plyr             1.8.9      2023-10-02 [1] CRAN (R 4.1.3)
##     pmplots        * 0.3.7      2023-11-04 [1] MPNDEV (R 4.1.3)
##     pmtables       * 0.6.0      2023-11-04 [1] MPNDEV (R 4.1.3)
##     posterior      * 1.4.1      2023-03-14 [1] CRAN (R 4.1.3)
##     prettyunits      1.2.0      2023-09-24 [1] CRAN (R 4.1.3)
##     processx         3.8.2      2023-06-30 [1] CRAN (R 4.1.3)
##     progress         1.2.2      2019-05-16 [1] CRAN (R 4.1.3)
##     progressr        0.14.0     2023-08-10 [1] CRAN (R 4.1.3)
##     ps               1.7.5      2023-04-18 [1] CRAN (R 4.1.3)
##     purrr          * 1.0.2      2023-08-10 [1] CRAN (R 4.1.3)
##     QuickJSR         1.0.7      2023-10-15 [1] CRAN (R 4.1.3)
##     R6               2.5.1      2021-08-19 [1] CRAN (R 4.1.3)
##     RColorBrewer     1.1-3      2022-04-03 [1] CRAN (R 4.1.3)
##     Rcpp             1.0.11     2023-07-06 [1] CRAN (R 4.1.3)
##     RcppParallel     5.1.7      2023-02-27 [1] CRAN (R 4.1.3)
##     readr          * 2.1.4      2023-02-10 [1] CRAN (R 4.1.3)
##     renv             1.0.3      2023-09-19 [1] CRAN (R 4.1.3)
##     reshape          0.8.9      2022-04-12 [1] CRAN (R 4.1.3)
##     reshape2         1.4.4      2020-04-09 [1] CRAN (R 4.1.3)
##     rlang            1.1.1      2023-04-28 [1] CRAN (R 4.1.3)
##     rmarkdown        2.25       2023-09-18 [1] CRAN (R 4.1.3)
##     rprojroot        2.0.3      2022-04-02 [1] CRAN (R 4.1.3)
##     rstan            2.32.3     2023-10-15 [1] CRAN (R 4.1.3)
##     rstudioapi       0.15.0     2023-07-07 [1] CRAN (R 4.1.3)
##     sass             0.4.7      2023-07-15 [1] CRAN (R 4.1.3)
##     scales           1.2.1      2022-08-20 [1] CRAN (R 4.1.3)
##     sessioninfo      1.2.2      2021-12-06 [1] CRAN (R 4.1.3)
##     StanHeaders      2.26.28    2023-09-07 [1] CRAN (R 4.1.3)
##     stringi          1.7.12     2023-01-11 [1] CRAN (R 4.1.3)
##     stringr        * 1.5.0      2022-12-02 [1] CRAN (R 4.1.3)
##     tensorA          0.36.2     2020-11-19 [1] CRAN (R 4.1.3)
##     tibble         * 3.2.1      2023-03-20 [1] CRAN (R 4.1.3)
##     tidyr          * 1.3.0      2023-01-24 [1] CRAN (R 4.1.3)
##     tidyselect       1.2.0      2022-10-10 [1] CRAN (R 4.1.3)
##     tidyverse      * 2.0.0      2023-02-22 [1] CRAN (R 4.1.3)
##     timechange       0.2.0      2023-01-11 [1] CRAN (R 4.1.3)
##     tzdb             0.4.0      2023-05-12 [1] CRAN (R 4.1.3)
##     utf8             1.2.3      2023-01-31 [1] CRAN (R 4.1.3)
##     vctrs            0.6.4      2023-10-12 [1] CRAN (R 4.1.3)
##     vroom            1.6.4      2023-10-02 [1] CRAN (R 4.1.3)
##     withr            2.5.0      2022-03-03 [1] CRAN (R 4.1.3)
##     xfun             0.40       2023-08-09 [1] CRAN (R 4.1.3)
##     xtable           1.8-4      2019-04-21 [1] CRAN (R 4.1.3)
##  VP yaml             2.3.5      2023-01-23 [?] CRAN (R 4.1.3) (on disk 2.3.7)
##     yspec          * 0.6.1      2023-11-24 [1] MPNDEV (R 4.1.3)
## 
##  [1] /data/NMBayesTutorial/renv/library/R-4.1/x86_64-pc-linux-gnu
##  [2] /data/home/timw/R/renvExt
##  [3] /data/home/timw/.cache/R/renv/sandbox/R-4.1/x86_64-pc-linux-gnu/dbd573bd
## 
##  V ── Loaded and on-disk version mismatch.
##  P ── Loaded and on-disk path mismatch.
## 
## ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
bbr::bbi_version()
## [1] "3.3.0"